PP-11: convert, named-axis OME-TIFF copies of images with axis errors - #124

PP-11 of the v0.4.0 plan: design v0.4 §5, "Conversion, the exception path". Version as main (0.4.0rc3, merged from main).

Summary (by module)

• conversion (orchestration, new): convert_experiment(experiment_root, mapping, destination, exclude=, missing_as_singleton=, memory_limit=, progress=) and plan_conversion.
  • It first checks the experiment with the confirmed mapping, reading headers only. Any error other than the axis errors stops the conversion before anything is written. So do 64-bit integer images (OME-TIFF has no such type), planes that cannot be read within the memory limit even in blocks (check_planes), not enough free space, and settings that are not allowed (ValueError).
  • Images with axes_yx_swapped, axes_contradict_header, axes_x_before_y or axes_not_named are written as named-axis OME-TIFF with the user's roles. Every other image is copied byte for byte: images whose axes are already named and agree, OME-Zarr stores with their labels, and excluded images (never read).
  • The copy tree goes in a folder the user names (--to, required). It must be outside the experiment and must not contain it.
  • The copy tree's lock is held throughout. Each copy is written as <name>.partial, with the source's fingerprint checked before and after. All copies move into place together, then the record is written.
  • Any failure or interruption removes the .partial files, the copies already moved and the folders made, so the folder is left as it was.
  • On a rerun, a copy is reused only when its source, roles, copy axes, pixel sizes and the copy's own fingerprint are as recorded. New images are added. Anything else is refused (copy_differs, copy_folder_not_empty) and never overwritten. When every copy is reused, the record is kept as it was.
• conversion_issues (pure, new): the conversion's messages.
• copy_axes (pure, new): each copy's axes (every role in source order, then Y and X), its shape and its OME metadata.
  • Pixel sizes follow the roles; a relabelled axis gets an unknown scale, and the record says why.
  • A Y or X relabelled as the other (a Y/X swap, X before Y) keeps the header's size only when the header gives X and Y exactly equal (owner decision). Sizes are never exchanged, and a z-step is never written as a time step.
  • Channel names are kept only when C keeps its role.
• ome_tiff_copy (io, new): an uncompressed OME-TIFF laid out with tifffile. Planes come from the run's own iter_planes, in blocks over the memory limit, each block written at its offset. BigTIFF is chosen when the pixels plus 1 KiB per page would pass 4 GiB less 32 MiB.
• conversion_record (io, new): .morphobricks-conversion.json, plus review_conversion for the run's warnings.
• Run: on a copy tree, the manifest records conversion_record_sha256, the report has a conversion entry, and every row is recomputed. The warnings conversion_source_changed and copy_changed_after_conversion are new; an unreadable record is conversion_record_unreadable.
• RunInputs.open refuses a run whose conversion record has changed or disappeared since (owner decision).
• CLI: convert EXPERIMENT --to FOLDER --axes ROLES with --exclude, --missing-as-singleton, --memory-limit, --remove-stale-lock and --log. It prints the next check and run commands.
• Messages: the axis-error actions, axis_check_summary and the notebook's axis marks name the convert command.
• Docs: contract §17, plus updates to §8, §9, §9.1 and §10; plan §10 (PP-11, plus PP-11b next) and question 59 (decided); user guide; README; AGENTS.md.

Owner decisions (2 November 2026, plan question 59, all decided)

1. Pixel sizes after a Y/X swap, or any Y or X relabelled as the other: carried to the copy when the header's X and Y are exactly equal (value and unit); otherwise both are unknown. Never exchanged.
2. --to stays required.
3. RunInputs.open checks conversion_record_sha256.
4. The notebook card is PP-11b, the next PR (plan §10).
5. The other interpretations (items 2 to 13) are accepted as built.

Criteria of the obsolete issues met here (so they can be closed)

• [Sad Catapella B3][Scientific correctness] OME-Zarr pyramid mean corrupts uint64/int64 precision #75: no 64-bit precision loss. int64 and uint64 are refused, and no arithmetic touches pixels: each copy plane is the read plane, bit for bit.
• [Sad Catapella B3][Memory] Blockwise pyramid region selector does not enforce a viable minimum budget #76: the memory budget fails fast. check_planes runs before anything is written, and planes are read and written in blocks within the limit.
• [Sad Catapella B3][Performance] TIFF-to-Zarr conversion rereads the converted pixels to compute histograms #79: the run reads the copies themselves, so the table describes the persisted pixels. The end-to-end CSV is identical.
• [Sad Catapella B3][Validation] ConversionSettings accepts noninteger numeric inputs that fail downstream #80: strict settings. memory_limit must be an int above 0 and not a bool, and missing_as_singleton must be a bool. Both are checked before any I/O.
• [Sad Catapella B5][Scientific integrity] Reused TIFF-converted Zarr copies are not validated against their stored pixels #87, [Sad Catapella B5][Correctness] Channel-label copy freshness check accepts missing or truncated OMERO metadata #88: before a copy is reused, its fingerprint (size, mtime) is checked against the record. Each run re-reads the copies by their fingerprints.
• [Sad Catapella B6][Scientific calibration] Reused OME-Zarr copies are not checked for numeric pixel-scale correctness #92: the copy's scales are checked before reuse. The recorded pixel sizes and unknown sizes must equal what the rules give now.
• [Sad Catapella B6][Performance] TIFF-to-Zarr conversion reopens headers to read physical calibration already inspected #96: no mixing of metadata epochs. The layout is taken from the checked header, the source fingerprint is checked before and after each copy, and a copy written under another rule is never reused next to new ones.

Pixel-exactness evidence

• The tests compare every plane of each copy with the original read under the confirmed roles, byte for byte, with dtype and shape. Cases: plain stacks, ImageJ slice stacks and hyperstacks, Y/X-swapped OME, multi-channel, CYX confirmed as ZYX, X before Y, RGB swapped, int8/uint16/int32/float32 with NaN and ±inf, big-endian float, block reads, BigTIFF.

• The independent review re-checked the following, all byte-identical:

  • an ImageJ TZCYX hyperstack, Y/X swapped;
  • a big-endian float32 OME with NaN and ±inf, Y/X swapped (the copy is little-endian with identical bits);
  • a plain stack, X before Y;
  • an OME ZCYX int16 with 3 named channels, swapped, read in blocks;
  • a tiled float32 plain stack read in blocks;
  • an RGB YXS stack, swapped.

  The copies pass check with axes_from: "header" and no errors, and the originals' SHA-256 are unchanged.

• End to end: a plain multi-channel float experiment with a design was run, converted, then run on the copies with the same output folder. plane_histograms.csv is byte-identical, and RunInputs reads the copies.

• 256 MiB uint16 (8 × 4096 × 4096), Y/X swapped:

  Memory limit	Time	Peak RSS	Identical planes
  1 GiB	2.9 s	158 MiB	8 of 8
  64 MiB	1.9 s	75 MiB	8 of 8

Tests

• Full local gate (ruff check, ruff format --check, tools/check.py --rules, pytest with .[dev,notebook,zarr], Python 3.12): 1750 passed, 1 skipped. The skip is test_discovery.py:665, expected when running as root. Main has 1691 passed, 1 skipped.
• 59 new tests in tests/test_convert.py and tests/test_convert_run.py. The review's regression tests fail on the PR's previous head.

§9 figures

• New files, in lines: conversion 587, conversion_record 386, copy_axes 247, ome_tiff_copy 152, cli_convert 141, conversion_issues 136.
• Largest new function: cli_convert.add_convert_parser, 48 executable lines.
• Highest complexity: conversion._write_all, 8.
• Most parameters: convert_experiment, 7.
• Ruff C901/PLR0913/PLR0915/ANN pass, the baseline is unchanged, and the # noqa count is unchanged (49).

Not verified: real microscopy files (simulated data only).

https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc

Generated by Claude Code